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ABSTRACT 



At a community college in Hawaii, two separate validation 
processes were carried out in order to install a new student placement test, 
each serving a different purpose at different times in the installation 
process. Decision Theory was used to arrive at a decision tool to allow for 
compromises between accuracy and enrollment management. The American College 
Testing Program's COMPASS test was chosen as the placement test for English 
and mathematics. A linear regression equation, based on responses of 400 
students, was generated for each COMPASS test to allow the conversion of 
current placement scores to COMPASS scores to permit a smooth transition in 
test use. Because the new test replicated the supposed accuracy of the old 
test, validity was studied with the Decision Theory approach. An alternative 
approach used to solve some of the validation dilemmas consisted of using 
instructors' opinions about the correctness of student placement in their 
courses at the beginning of the semester. However, this approach proved to be 
of little use. (SLD) 
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Introduction 

About four years ago, this college decided to use the American College Testing (ACT) Service's 
COMPASS as its placement test for both English and math. Various steps were needed to install this 
computer-based test on campus in a manner that (a) does not initially disturb the prevailing course 
population significantly and (b) ultimately leads to improved accuracy in placing students. 



Methodology 

Although validation data were provided by ACT, it was decided that this college will generate its 
own validation data instead of using ACT's data. The following steps were taken for the installation 
of a new placement test: 

1 . Initial Validation: Maintaining the Status Quo 

Initially, a campus wanting to validate locally a new placement test is typically faced with the lack 
of data to determine optimal cut-off scores. One method that can be used to generate 
temporary data that has been used at this college is based on the principle of maintaining the 
same course population as the one that exists with the current test. To accomplish this, 
COMPASS was administered to a random sample of new enrollees who had taken the current 
placement test only a few weeks earlier. Based on 460 students, a simple linear regression 
equation was then generated for each COMPASS sub-test. For this college and for the Reading 
sub-test, for example, it was: 

y = 24.0576 + 4.1420 x, where y is the COMPASS score, and 

x is the score on the current test (the Nelson-Denny). 

Once derived, this set of equations allow the conversion of the current placement scores to the 
COMPASS scores and allows for a smooth transition from the current placement test to the new 
placement test with little or no shift in course population. 

The problem inherent in this approach is that, theoretically, the new test replicates the same 
accuracy level that the current placement test has. In case of this college, the accuracy of the 
Nelson-Denny had never been studied. Yet, it was expected that the use of equivalent cut-off 
scores on COMPASS would not improve placement accuracy, which did not allow the college to 
realize one of the initial incentives to adopt COMPASS. 
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2. Second Valldadon: Cut-Off Score Revision 



The second validation process should be carried out after allowing time to generate enough data 
for a more encompassing study. The methodology involved in this phase relied heavily on 
correlation and regression techniques, using the final grade on the first English or math course 
immediately following the placement as the dependent variable. The methodology consists of 
two separate steps, the first is the calculation of validity measures, and the second one is to 
generate new cut-off points. 



A. Validation: The Use of Logistic Regression 

Because the dependent variable, namely success in the particular course in which the student was 
placed, is dichotomous (pass or fail outcomes only), logistic regression was conducted to 
calculate validity. The results of the regression for this college are shown in Table 1 . 



Table 1 

COMPASS Logistic Regression Analysis Results 
p-Value for Wald Statistic 
(Sample Size) 



Course 


Reading 


Writing 






Eng A 


0.0496* 


0.0009* 








(131) 


(132) 






Eng B 


0.5086 


0.1150 








(171) 


(171) 






Eng C 


0.9814 


0.0014* 








(413) 


(413) 






Eng D 


0.9755 


0.2546 








(565) 


(565) 






EngE 


0.3134 


0.9456 








(31) 


(30) 
















Course 


Pre-Alg 


Alg 


Coll Alg 


Trig 


Math A 


0.1847 


0.2959 


X 


X 




(397) 


(48) 






Math B 


0.0000* 


0.0006* 


X 


X 




(412) 


(353) 






Math C 


0.0159* 


0.0036 * 


X 


X 




(215) 


(224) 






Math D 


0.2637 


0.0685 * 


X 


X 




(17) 


(22) 







* Statistically significant measure 
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The data indicated that: 



1 . Math was a better predictor of subsequent math course performance than English was for 
subsequent English courses. 

2. The validity of the sub-tests making up the English and math tests was quite varied. In 
general, writing was a better predictor of performance in English courses than reading, and 
pre-Algebra and Algebra were better predictors of their corresponding courses than were 
College Algebra and Trigonometry (not shown in the table for other Math courses). The 
superior predictive ability of writing over reading supports the practice at several higher- 
education institutions where writing samples are used to place students (Schum, 1985). 



Table 2 

Course-Specific Decision Analysis 
for COMPASS Cutoff- Score Determination* 





States of Nature** 




Decision Alternatives 


J ■ < ‘ •%; - H : * Success In Course 


.! &W'-.,-No Success Iri^Courae^M? 


Admit 


A 


B 


Do Not Admit ' •‘v. v: 






Statistical Indices 

For each COMPASS score in each course, the following three indices were calculated: 


Percent Not Admitted 
Accuracy Rate = 
Success Rate 


= (C+D)/(A + B+C+D)xlOO 

(A + D)/(A+B + C+D) 

= A/(A + B ) 





* Adapted from Noble and Sawyer (I 997) 

* * Situations not under the control of the decision maker 



B. New Cut-Off Score Derivation: The Use of Decision Theory 

In addition to logistic regression, Decision Theory was used for the calculation of 3 rates: 
accuracy rates, success rates, and enrollment rates. The formulas to calculate these indices are 
presented in Table 2, and the results in Table 3. Table 3 shows the example of one math 
course and one COMPASS sub-test, in this table, the framed number indicates the current cut- 
off. A similar set of calculations needs to be derived for each course and sub-test used for 
placement purposes. 

The data in Table 3 provide a powerful decision-making tool which allows for compromises 
between rigid statistical decisions and enrollment-management considerations. 
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Table 3 

Placement Results for Hath C - Pre-Algebra Sub-test Scores 



COMPASS 

Score 


% 

Not Admit 


Accuracy 

Rate 


Success 

Rate 


23 


2.44 


0.30 


0.40 


24 


4.88 


0.47 


0.47 


30 


9.76 


0.40 


0.50 


33 


12.20 


0.43 


0.53 


35 


14.63 


0.39 


0.49 


36 


17.07 


0.45 


0.45 


38 


24.39 


0.48 


0.48 


39 


29.27 


0.51 


0.51 


40 


31.71 


0.56 


0.56 


44 


34.15 


0.53 


0.53 


50 


39.02 


0.55 


0.55 


51 


43.90 


0.57 


0.57 


52 


48.78 


0.58 


0.58 


54 


51.22 


0.56 


0.56 


55 


58.54 


0.56 


0.56 


56 


60.98 


0.55 


0.59 


59 


63.41 


0.53 


0.63 


61 


65.85 


0.52 


0.62 


62 


75.61 


0.48 


0.68 


70 


78.05 


0.40 


0.70 


72 


80.49 


0.32 


0.72 


76 


82.93 


0.34 


0.74 


79 


85.37 


0.33 


0.73 


81 


90.24 


0.34 


0.79 


85 


92.68 


0.36 


0.86 


89 


95.12 


0.35 


0.85 


91 


97.56 


0.37 


0.87 



C Limitations 

In studies of this type, the following dilemmas are usually encountered: 

1 . The calculation of predictive validity of placement tests when final course grades are used as 
the dependent variable is conceptually flawed, since placement test scores are designed to 
place students in the beginning of a course. Inherent in this approach is the incorrect 
assumption that a student who places correctly in a certain course is expected to pass this 
course. Thus, extraneous emotional, family- or job-related factors are not considered 
(Abou-Sayf, 2000A). 

2. Grades are inherently poor as criterion variables. 

3. In multi-campus districts or systems such as this one, articulation imposes the use of uniform 
cut-offs for all campuses, regardless of the difference in ability of the applicants. 
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Alternative Validation Measures 



As an alternative to using final grades as the dependent variable, instructors' opinion about whether 
each student in their class has been appropriately placed based on their knowledge of the student's 
ability was sought early on in the semester (Abou-Sayf, 2000B). The use of surveys to validate 
placement test results is not novel. Armstrong and Takahata (1991 ), for example, used results of 
faculty and students' opinions on the accuracy of the test as the dependent variable. In this study, 
data on 2,856 students in 15 courses, 5 English and 10 Math courses, were obtained from the 
instructors and correlated with placement tests. The resulting phi coefficients are presented in Table 
4 and indicate that there was a low agreement between the placement test results and the 
instructors' evaluation of the adequacy of the student's placement. The results also indicate that 
agreement between these two measures was even weaker at the lower-level classes than at the higher 
ones. 



Table 4 

COMPASS Phi Coefficients 



English 


Math 


Course 


Correlation Coefficient 
Phi 


n 


Course 


Correlation Coefficient 
Phi 


n 


An 


0.205489329 


570 


All 


0.399969282 


858 


Eng A 


0.149786172 


61 


Math A 


0.384514959 


40 


Eng B 


0.137940147 


87 


Math B 


0.355473344 


151 


EngC 


0.324698847 


120 


Math C 


0.421350486 


170 


Eng D 


0.185511996 


302 


Math D 


0.289157466 


36 








MathE 


0.396964883 


285 








Math F 


0.262346929 


60 








Math G 


0.452620765 


81 








Math H 


0.471404521 


20 








Mathl 


0.480384461 


15 



Another alternative to the use of final exam grades as the dependent variable would be to use the 
scores on a competency-based exam to all students that would be administered at the very beginning 
of the course, more research is needed in this area. 



Summary and Conclusions 

At this college, two separate validation processes were carried out in order to install a new 
placement, each serving a different purpose at a different point in the installation process. Decision 
Theory was used to arrive at a decision tool that allows for compromises between accuracy and 
enrollment management. An alternative approach used to solve some of the validation dilemmas 



and consisting of using instructors' opinion about the correctness of student placement in their 
courses at the beginning of the semester proved to be of little use. 
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